-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BigQuery: Add to_standard_sql() method to SchemaField #8880
Conversation
|
||
# NOTE: No need to also handle the "ARRAY" composed type, the latter | ||
# does not exist in legacy SQL types. | ||
if sql_type.type_kind == types.StandardSqlDataType.STRUCT: # noqa: E721 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
False positive here, flake8
thinks that is about comparing types and wants to enforce isinstance()
- but that would not work, as we actually compare integers here.
Support for standard SQL type names and simple arrays added. Still need to add support for arrays containing STRUCTs, do not merge yet. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. Thanks for adding tests for arrays of structs.
This reminds me that I wrote some similar code recently at
def bq_to_arrow_data_type(field): |
Would it make any sense to share any of that code here?
@tswast You mean sharing the conversion logic or the test for it? Since the "target results" differ, there are differences between the two implementations, and they are both reasonably short, too, thus we might not actually gain that much by refactoring. (but of course, if you see something "obvious" that would represent a good opportunity, we can add this improvement in another PR) |
Agreed. Not obvious to me right now how to share code without it getting more complex. |
Closes #8494.
This PR adds a method to convert a legacy table field instance to the new
StandardSqlField
type.How to test
Verify that the types are correctly mapped from the legacy to standard SQL, and that the new method does what it is supposed to do.